In the field of human interaction recognition, local features based on RGB video often cannot effectively distinguish approximate actions. The Depth image information and the color image information are merged in the recognition process, and a two⁃person interactive behavior recognition algorithm that integrates the depth information and the individual segmentation fusion is proposed.The algorithm firstly extracts the points of interest for RGB and Depth video, then uses 3DSIFT to describe the features on RGB video. The YOLO network is introduced into divide the left and right points of interest on the Depth video, and the local co⁃occurrence matrix is used for local correlation information description. Finally, the nearest neighbor classifier is used to classify the RGB features and Depth features, and further the recognition results are obtained by the decision⁃level fusion, which improves the accuracy of recognition. The results show that the combination of depth visual co⁃occurrence matrix can greatly improve the recognition accuracy of double interaction behavior, and the correct recognition rate of 90% of the actions in SBU Kinect interaction database can verify the effectiveness of the proposed algorithm.